50

6

The Nature of Information

text. ¹ Consider a would-be passenger racing up to a railway station. His question “has

the train gone?” may indeed be answered by “yes” or “no”—although, in practice,

a third alternative, “don’t know”, may be encountered. At a small wayside station,

with the traveller arriving within five minutes of the expected departure time of the

only train scheduled within the next hour, the answer (yes or no) would be unam-

biguous and will convey exactly one bit of information, as will be explained below.

If we insist on the qualification “desired”, an unsolicited remark of the stationmaster,

“the train has gone”, may or may not convey information to the hopeful passenger.

Should the traveller have seen with his own eyes the train depart a minute before,

the stationmaster’s remark would certainly not convey any information.

Consider now a junction at which, after leaving the station, the lines diverge in

three different directions. The remark “the train has gone”, assuming the information

was desired, would still convey one bit of information, but by in addition specifying

the direction, viz., “the train has gone to X”, or “the train to X has gone”, “X” being

one of the three possible destinations, the remark would convey log Subscript 2 Baseline 3 equals 1.59log2 3 = 1.59 bits

of information, this being the average number of questions admitting yes/no answers

required to specify the fact of departure to X, as opposed to either of the two other

directions.

This little scenario illustrates several crucial points:

1. Variety exists. In a formless, amorphous world there is no information to convey.

2. The amount of information received depends on what the recipient knows already.

3. The amount of information can only be calculated if the set of possible messages

(responses) has been predefined.

Dichotomous information often has a hierarchical structure; for example, on a

journey, a selection of direction has to be made at every cross-road. Given an ulti-

mate destination, successive choices are only meaningful on the basis of preceding

ones. Consider also an infant, who “chooses” (according to its environment) which

language it will speak. As an adolescent, he chooses a profession, again with an

influence from the environment and, in making this choice, knowledge of a certain

language may be primordial. As an adult there will be further career choices, which

will usually be intimately related to the previous choice of a profession.

Let us now reexamine the measurement of the length of a stick. It must be specified

in advance that it does not exceed a certain value—say one foot. This will suffice to

allow an appropriate measuring tool to be selected. If all we had was a measuring

stick exactly one foot long, we could simply ascertain whether the unknown piece

was longer or shorter, and this information would provide one bit of information, if

any length was a priori possible for the unknown piece.

Suppose, however, that the measuring stick is marked off in 1-inch divisions. If

the probabilities pp of the unknown piece being any particular length ll (measured to

the nearest inch), with0 less than l less than or equals 120 < l ≤12, were a priori equal (i.e.,p equals one twelfthp =

1

12 ^{for each possible}

length), then the information produced by the measurement equals log Subscript 2 Baseline 12 equals 3.59log2 12 = 3.59

1 When it comes to the quantification of information, context is usually formalized through the

provision of a finite set of possible answers (choices). See Sect. 6.3.2.